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Digital Ephemera Collections 



• Collected by researchers 

• Donated by activists 

• Images, audio, video, scanned documents, social media, 
web server logs 




NewsScape 



>244,000 hours of TV news archived 
Recorded 2005-present 
13 countries, 9 languages 
38 networks 

Searchable by captions, on-screen text, named entities 
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AUazeera News 



BEGAN JOINS US AT NEW YORK TIME SQUARE . 
PAUL? » Reporter: THATS RIGHT, JOHN. WE 
ARE ABOUT TWO TO THREE BLOCKS WEST Or 
TTHE5 SQUARE, THE CORNER OF LDth AVENUE 
AND 43 nd STREET. A LARGE G ROUP OF 
PROTESTERS JUST CROSSED ON THEIR WAr 
BACK TO TIM E SQUARE. TH EY HAVE BEEN 
DISRUPTING TftAFFJC AjfcOuND THIS AREA. WE 
FOLLOWED THEM OUT Of TIMES SQUARE A 
LITTLE WHILE AGO. ABOUT ISO TO 200 PEOPLE 
WHO LAID DOWN IN THE Mt^" '"'THE 
INTERSECTION, RIGHT jn Fkun. ur p'HE I 'OK I 
AUTHORITY 8US TERMINAL RIGHT tN THE HEART 
OF TIMES SQUARE, BLOCKED TRAFFIC FOR 
THREE TO FOUR MINUTES, WE JUMPED OUT OF 
THE VEHICLE. FOLLOWED THEM WEST DOWN 
12nd STREET. I THINK THET WENT TO THE WEST 
SIDE HIGHWAY WHERE THERE IS ANOTHER 
GROUP Or LARGE PROTESTERS TAR ARER 
UPTOWN DISRUPTS fi TRAGIC. 3T LOOKS LTKE 
TtlEY MADE A LOOP AND ARE HEADING BACK 
TO WARDS TIMES SQUARE. OltS COULD HO ON 
ALL MIGHT, THT5GAMC Of CAT AND MOUSE 
WCTH THE PROTESTERS. THEY ARE BETNG 
FOLLOWED EVERYWHERE THEY GO BY A VERY 
LARGE POLICE PRESENCE BUT THE POLICE ARE 
STANDIMG OFF, REALLY LETTING TilEM HAVE 
THEIR RUN OF THE STREETS, NOT OBSTRUCTED 
THEM IN ANY WAV. JUST SORT OF REALLY 



sho. 
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WWW Russia Today 



opinions on this story. Tell us how you think the 
Strait of Hormuz deadlock will evolve. About half of 
those who voted so far think the U.S. will use the 
standoff as a pretext to attack Iran. Just over a 
third say it won't have immediate consequences, 
being just the latest move In a long game. Only 8 
percent think Iran will teach the West a lesson by 
cutting off vital supplies. And even fewer think 
Tehran will achieve its end goal of preventing 
sanctions. Cast YOUR vote at RT dot COM. Coming 
up later in the programme: the final countdown ... 
RT brings you the view from ins de the Occupy 
movement as we show it from a new, up close and 
personal perspective ... Egyptian rights groups 
have accused military rulers of using what they 
describe as Mubarak-era tools of repression. That's 
after soldiers and police raided the offices of ten 
non governmental organisations, including some 
foreign ones. RT's Mana Finoshina has more. 
Foreign policy analyst Dr Adil Shamoo believes the 
military government is making sure it stays in 
power, but the people are determined for 
democratic change. And coming up after 3:30 PM 
GMT, the gjests on Crosstalk discuss how the 
Egyptian revolution in February set a trend for 
similar movements across the globe... And inspired 
by the Egyptian revolution, opposition grojps in 
Syria are calling for large-scale anti-regime 
marches across the country. That's amid reports of 
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How SoLoGlo is different 



I. Real-time capture and archiving of tweets, 
referenced URIs, and embedded resources 

II. Rapid analysis, real-time opportunities 



. Collection-agnostic linking 



Django administration 



Home > Ul > Twitter filters 

O The twitter filter '5* was added successfully. 

Select twitter filter to change 



archive, tod ay 

webpape capture book mark let; Drag this button ana 


fag btog ask me contact donate 
hive.taday to The tool ha r and you will be able to -'save pages in one click! 


My url is alive and I want to archive Its content. 


pi tp : fj www. d Dmain.com/url 


submit url 
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Twitter's Contribution 



• Twitter has made full 
archive of tweets available 

• Indexed, searchable 

• Not accessible via API 

• How about deleted tweets? 

• Real-time capture of 
embedded resources? 



Building a complete Tweet index 

Tuesday, November IE, 2014 I By Yi Zhuang (@yz) [17:35 UTC] 



Today, we are pleased to announce that Twitter now indexes 
every public Tweet since 2006, 

Since that first simple Tweet over eight years ago, hundreds of billions of Tweets have 
captured everyday human experiences and major historical events. Our search engine 
excelled at surfacing breaking news and events in real time, and our search index 
infrastructure reflected this strong emphasis on recency. But our long-standing goal has 
been to let people search through every Tweet ever published. 

This new infrastructure enables many use cases, providing comprehensive results for 
entire TV and sports seasons, conferences (STEDGlobal), industry discussions 
(#MobllePayments), places, businesses and long-lived hashtag conversations across 
topics, such as # Japan Earthquake, #Election2012, #ScotlandDecides, # Hong Kong. 

# Ferguson and many more. This change will be rolling out to users over the next few 
days. 

In this post,, we describe how we built a search service that efficiently indexes roughly 
half a trillion documents and serves queries with an average latency of under 100ms. 

The most important factors in our design were: 

■ Modularity: Twitter already had a real -time index (an inverted index 
containing about a week's worth of recent Tweets}. We shared source code 
and tests between the two indices where possible, which created a cleaner 
system in less time. 

* Scalability: The full index is more than 1 00 times larger than our real-time 
index and grows by several billion Tweets a week. Our fixed -size real-time 
index clusters are non -trivial to expand; adding capacity requires re- 
partitioning and significant operational overhead. We needed a system that 



https://bloq.twitter.com/2014/buildinq-a-complete-tweet-index 
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SoLoGlo Collections 1/3 



• Twitter dataset about Egyptian revolution 

>400k tweets 
50k unique users 

Tweets originated from within 200 miles around Cairo 

25% of tweets contain references to external resources (web 
pages, images, videos, etc) 



JSSm i ©tamerbadereldin 

"©NevineZaki: A pic I took yesterday of Christians 
protecting Muslims during their prayers #jan25 
http://yfrog.com/h02gvclj" 

9:47 AM -6 Feb 2011 

4% «- ★ 
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SoLoGlo Collections 1/3 



Twitter dataset about Egyptian revolution 

>400k tweets 
50k unique users 

Tweets originated from within 200 miles around Cairo 

25% of tweets contain references to external resources (web 

pages, images, videos, etc) v** yf rog ges - 



q, Upload 



20% of references are dead 



Flights to Panama 
City 

Fly to Panama City, Panama 
and Get Our Guaranteed 
Lowest Fare. 
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SoLoGlo Collections 1/3 



• Twitter dataset about Egyptian revolution 

>400k tweets 
50k unique users 

Tweets originated from within 200 miles around Cairo 

25% of tweets contain references to external resources (web 
pages, images, videos, etc) 

20% of references are dead & .^^^^ i*™- 

i£Jjjjl j ©tamerbac s 

60% Of these are nOt archived "@NevineZaki: A pic I tookjJPerday of Christians 

protecting Muslims durinc/_Jeir prayers #jan25 
http://yfrog.com/h02gvcljD 

9:47 AM -6 Feb 2011 

4* «- ★ 
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SoLoGlo Collections 1/3 



This one 
is! 




W^Sf , ou , images and video on twitter 


<\ > ] EH 


Upload Popular Mobile Apps My yfrog 


1 ■ i, 






Views; 



Rating; 



f i 

26.7K Share: 
Tweet 



Face book Twiner Myspace Digg Reddit Email 



http://wavback.archive-it.org/all/2011 0203083908/http://vfroq.com/h02qvcli 
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Rapid analysis 



#keystoneXL 
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[Courtney J RT @TheTyee Check om Tyee coninbutmg. ednor AinHew N*itoruk's op-ed on Keystone XL in New York Times http 01 eo.'8|LloDfso] ffKeysloneXL 
[Stive Kamp] RT @TneBax1er&egn OVER 300 PER YEAR New Analyst Shows Devastating impact oi U 5 Pipelm* Spills fftoysloneXL http -1 co-fnlUcXinkl h(tp 
[Fiatfh^issl RT Qdebreese Mediaite headline tomlere keystone ts wrong I fixed i1 *KXL «*KeystoneXL g!Medie«te htfp 01 coSGaShYFltTi 
[Emma Green] RT QHaipef The Hulk HULKHARPER NO GET PIPELINE HULKHARPER FEEL EMPTY INSIDE NOW HULKHARPER NO LOOK FORWARD TO 
CALL FROM KOCH BROTHER atdnp 

[WiHoughby B Dobb* ] RT @-mashawe TMe Senate has defeated a bill to approve me controversial HKeystoneXL pipeline fittp it\ co-nBVOFOUiay 
http :<\ cd'AiqttiiVp 

[blackbutterTIyJ RT (gPfjterKelteet I am happy Ihet the US ^Senate defeats PKeystoneXl pitwhrw t>y one vOite lodeyi hKp >A C0'ROpivWG86k via @usa1od£iy pK© 
[Grey Goo| Wind and solar power ere even more expense than is commonly thought nttp W co ; 'gtrpLv:sf7 #KeystoneXL *tcoi 



SoLoGlo 

Martin Klein, Peter Broadwell, Todd Grappone, Sharon Farb 
#PDA2015, NYC, April 25th 2015 



UCLA LIBRARY 



SoLoGlo Collections 3/3 

• Collection on AirAsia QZ8501 crash on 12/28/2014 

7.3 million tweets containing #AirAsia or #QZ8501 
1 .3 million distinct users 
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• Collection on AirAsia QZ8501 crash on 1 2/28/201 4 

7.3 million tweets containing #AirAsia or #QZ8501 
1 .3 million distinct users 
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SoLoGlo Collections 3/ 
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SoLoGlo Collections, Encore 
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Ed Summers 

@edsu 



Following 



can't make it to #pda2015? follow along at 

inkdroid.org/pda2015/ to see what URLs 

people are tweeting about #pda20i5 uris 



9 College Park, MD 



15 

13 
7 
7 
6 



Annotated bibliography for my talk at Personal Digital Archiving 
201 5 « Eira Tansey 



http://inkdroid.org/pda201 5/ 
https://twitter.com/edsu/status/591 681 640801 37421 1 



1World1 Family. me 
Preservation Access Program 

& a- M A <■ 2» r 

PDA Conference 2015 | #PDA2015 

#The Finding Aid 
Jessica Bushey 

mmmm 
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Archived tweets, URIs, and embedded media for #pda2015 



Date 


Tweet ID 


Embedded URI 


Embedded Media 


22T1 1:40:39 


590842648501624832 T 




http ://pbs .twimg .com/m . . . 


901 S 04 

22T1 1:42:51 


590843204544700416 T 




http ://pbs .twimg .com/m . . . 


901 04 

22T1 1:50:46 


590845194884538368 T 




http : //pbs .twimg .com/m . . . 


901 S 04 

22T11:51:24 


590845353445949440 , 




http : //pbs .twimg .com/m . . . 


90 1 ^ 04 

22T1 1:57:48 


590846965308645376 , 




http : //pbs .twimg .com/m . . . 


90 1 ^ 04 

22T14:15:55 


590881723086295040 , 


httD://ow.lv/KT4vO/^ 




901 04 

23T00:30:06 


591036287626436608 r 




http : //pbs .twimg .com/m . . . 


901 ^ 04 

23T11:13:47 


591198276113731584 T 


http ://ww w .neh . go v/ne w . . . 




901 ^ 04 

23T12:26:25 


591216553057738752 T 




http://pbs. twimg. com/m... 


2015-04- 
23T16:45:16 


591281698010243072 T 


http://Dersonaldigital . . . ^ 




2015-04- 
23T23:38:04 


591385582682648576 T 


http://personaldigital . . . 




2015-04- 
24T02:33:27 


591429715623280640 T 




http://pbs .twimg .com/m. . . 


2015-04- 
24T 11:50:24 


591569879637037056 T 


http://eiratansev .com/ . . . 




2015-04- 
24T1 1:53:07 


591570561081421824 T 


https://twitter.com/jo... T 





http://babvlon.librarv.ucla.edu/archived/pda2015.html 
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